The rapid adoption of large language models (LLMs) in healthcare has been accompanied by scrutiny of their oversight. Existing monitoring approaches, inherited from traditional machine learning (ML), are task-based and founded on assumed performance degradation arising from dataset drift. In contrast, with LLMs, inevitable model degradation due to changes in populations compared to the training dataset cannot be assumed, because LLMs were not trained for any specific task in any given population. We therefore propose a new organizing principle guiding generalist LLM monitoring that is scalable and grounded in how these models are developed and used in practice: capability-based monitoring. Capability-based monitoring is motivated by the fact that LLMs are generalist systems whose overlapping internal capabilities are reused across numerous downstream tasks. Instead of evaluating each downstream task independently, this approach organizes monitoring around shared model capabilities, such as summarization, reasoning, translation, or safety guardrails, in order to enable cross-task detection of systemic weaknesses, long-tail errors, and emergent behaviors that task-based monitoring may miss. We describe considerations for developers, organizational leaders, and professional societies for implementing a capability-based monitoring approach. Ultimately, capability-based monitoring will provide a scalable foundation for safe, adaptive, and collaborative monitoring of LLMs and future generalist artificial intelligence models in healthcare.

San Francisco-based Augmedix has progressed from a Google Glass-based clinical documentation startup to a publicly-traded, AI-enabled ambient automation platform that documents patient encounters and generates medical notes that can be transferred to an EHR. The company, founded in 2012, also provides pre- and post-visit documentation offerings to give providers a more complete digital picture of a patient's health journey. Ian Shakil, founder, director and chief strategy officer at Augmedix, spoke with MobiHealthNews to discuss the company's evolution and its anticipated release of a new product in 2023. MobiHealthNews: Can you tell me about Augmedix and how it works in the ambient documentation space? Ian Shakil: I started the company about 10 years ago with the mission to rehumanize the provider/patient interaction.